Day 14 Kubernetes 持久戰 - GlusterFS + Heketi 安裝基礎操作篇

本日重點與方向 (TAG): kubernetes、k8s、PV、persistent volume、PVC、persistent volume claim、Pod use PVC、GlusterFS、Heketi、SC、StorageClass
今天將會介紹使用 Bare Metal 進行 Kubernetes 環境中持久化儲存的軟體功能測試，主要會是以 GlusterFS (存儲管理)、Heketi (GlusterFS 操縱介面) 兩個開源專案進行驗證，並對於先前組建的 kubernetes 功能進行整合，如果你中途裝一裝有一些異常與失敗的話，基本上就是參考安裝筆記最下面的重建儲存叢集，還有搭配一下前幾天的重建 kubernetes 的做法，GlusterFS 配置上基本上就是的碟硬要乾淨一些(據說是要給整個分割區或硬碟)，Heketi 部分基本上不會出問題，也就是模板先改起來改一下帳號密碼的參數，之後就弄好啦，有副本需求相關的資源型態設定，他的設計的概念是基於 StorageClass 的配置，當然還有一些進階一點的去看官方網站 吧 (雖然官方已經 Archived 了) ，詳細的操作還是去看看輔助文章那邊的吧。

本文參考

https://blog.csdn.net/chengyuming123/java/article/details/86539986

本次使用設備資訊

Network Switch

• 數量: 1
• 型號: D-Link 1210-28 (L2 Switch)

Bare Metal

Master Node

• 數量: 1
• Ubuntu: 16.04 / 18.04
• Docker Version: 19.03
• CPU: E3-1230_V3 ^ 1
• RAM: 16GB
• OS_Disk: 120 GB (SSD)
• Data_Disk: 500 GB (HDD)
• Network: 1Gbps

Worker Node

• 數量: 2
• Ubuntu: 16.04 / 18.04
• Docker Version: 19.03
• CPU: E3-1230_v3
• RAM: 8 GB
• OS_Disk: 120 GB (SSD)
• Data_Disk: 250 GB (HDD)
• Network: 1Gbps

GlusterFS + Heketi on Kubernetes

環境需求

1. Ubuntu 18.04
2. Kubernetes 1.15.6 (Base on Docker)
3. 2 Disk (OS/Data)
4. 3 個 kubernetes 可用的部署節點 (master 可污染)

節點狀態 (設定參考)

處理節點上的資料碟等待備用

1. 觀看 Data_Disk 掛載點

• 指令

sudo fdisk -l

• 系統資訊
  

2. 清除 Data_Disk 上的資料

• 指令

sudo wipefs -a <Disk-Mount-Point>

• 強制清除

sudo wipefs -af <Disk-Mount-Point>

• Example

sudo wipefs -a
/dev/sdb

安裝 Heketi 環境

1. 下載 binary 檔 (版本)

wget https://github.com/heketi/heketi/releases/download/v10.0.0/heketi-v10.0.0.linux.amd64.tar.gz

3. 解壓縮/移至系統路徑下

tar -zxvf heketi-v10.0.0.linux.amd64.tar.gz heketi/
cp heketi/heketi /usr/bin/
cp heketi/heketi-cli /usr/bin/

安裝 GlusterFS 的客戶端 (DaemonSet Pod 中版本為 7.1)

1. GlusterFS Client 安裝

sudo add-apt-repository ppa:gluster/glusterfs-7
sudo apt-get update
sudo apt-get install glusterfs-client

2. 驗證安裝環境

• 指令

glusterfs --version

• 系統回饋

root@sdn-k8s-b2-1:~# glusterfs --version
glusterfs 7.7
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.

3. 修復 Glusterfs 功能 (ref)

• 測試環境指令

mount.glusterfs -V

• 系統異常回覆

root@sdn-k8s-b2-1:~# mount.glusterfs -V
/sbin/mount.glusterfs: 650: shift: can't shift that many

• 修復方式

nano /sbin/mount.glusterfs

• 修改內容

原內容
#!/bin/sh
修改後
#!/bin/bash

4. 修復不知名的錯誤 MDFK

• 後面 GlusterFS 的 DaemonSet Pod 起不來的問題)

rm -rf /etc/glusterfs /var/lib/glusterd

啟動 kernel 模組功能

1. 啟動模組

• 指令

sudo modprobe dm_thin_pool
sudo modprobe dm_mirror
sudo modprobe dm_snapshot

2. 檢查啟動的模組

• 指令

lsmod | grep dm_thin_pool
lsmod | grep dm_mirror
lsmod | grep dm_snapshot

• 回饋資訊

ubuntu@sdn-k8s-b2-1:~$ lsmod | grep dm_thin_pool
dm_thin_pool           69632  0
dm_persistent_data     73728  1 dm_thin_pool
dm_bio_prison          20480  1 dm_thin_pool

ubuntu@sdn-k8s-b2-1:~$ lsmod | grep dm_mirror
dm_mirror              24576  0
dm_region_hash         20480  1 dm_mirror
dm_log                 20480  2 dm_region_hash,dm_mirror

ubuntu@sdn-k8s-b2-1:~$ lsmod | grep dm_snapshot
dm_snapshot            40960  0
dm_bufio               28672  2 dm_persistent_data,dm_snapshot

Gluster 對主機 Port 使用

• 檢查指令

netstat -tupln | grep <port>

污染 kubernetes 節點成為 可部署 Gluster 的節點 (單節點污染)

kubectl taint nodes --all node-role.kubernetes.io/master-

在 Master 節點進行主服務安裝

下載 Gluster 官方實作 kubernetes 介接專案

git clone https://github.com/gluster/gluster-kubernetes.git

• 切入專案

cd gluster-kubernetes/deploy

修改 Gluster 部署 yaml 檔

nano kube-templates/glusterfs-daemonset.yaml

• 修改項目

# 註解
- "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi"
# 添加
- "systemctl status glusterd.service"

• 修改完成後的檔案

---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: glusterfs
  labels:
    glusterfs: daemonset
  annotations:
    description: GlusterFS DaemonSet
    tags: glusterfs
spec:
  template:
    metadata:
      name: glusterfs
      labels:
        glusterfs: pod
        glusterfs-node: pod
    spec:
      nodeSelector:
        storagenode: glusterfs
      hostNetwork: true
      containers:
      - image: gluster/gluster-centos:latest
        imagePullPolicy: IfNotPresent
        name: glusterfs
        env:
        # set GLUSTER_BLOCKD_STATUS_PROBE_ENABLE to "1" so the
        # readiness/liveness probe validate gluster-blockd as well
        - name: GLUSTER_BLOCKD_STATUS_PROBE_ENABLE
          value: "1"
        - name: GB_GLFS_LRU_COUNT
          value: "15"
        - name: TCMU_LOGDIR
          value: "/var/log/glusterfs/gluster-block"
        resources:
          requests:
            memory: 100Mi
            cpu: 100m
        volumeMounts:
        - name: glusterfs-heketi
          mountPath: "/var/lib/heketi"
        - name: glusterfs-run
          mountPath: "/run"
        - name: glusterfs-lvm
          mountPath: "/run/lvm"
        - name: glusterfs-etc
          mountPath: "/etc/glusterfs"
        - name: glusterfs-logs
          mountPath: "/var/log/glusterfs"
        - name: glusterfs-config
          mountPath: "/var/lib/glusterd"
        - name: glusterfs-dev-disk
          mountPath: "/dev/disk"
        - name: glusterfs-dev-mapper
          mountPath: "/dev/mapper"
        - name: glusterfs-misc
          mountPath: "/var/lib/misc/glusterfsd"
        - name: glusterfs-cgroup
          mountPath: "/sys/fs/cgroup"
          readOnly: true
        - name: glusterfs-ssl
          mountPath: "/etc/ssl"
          readOnly: true
        - name: kernel-modules
          mountPath: "/usr/lib/modules"
          readOnly: true
        securityContext:
          capabilities: {}
        - name: glusterfs-ssl
          mountPath: "/etc/ssl"
          readOnly: true
        - name: kernel-modules
          mountPath: "/usr/lib/modules"
          readOnly: true
        securityContext:
          capabilities: {}
          privileged: true
        readinessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 40
          exec:
            command:
            - "/bin/bash"
            - "-c"
            #- "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi"  //注释
            - "systemctl status glusterd.service"  //添加
          periodSeconds: 25
          successThreshold: 1
          failureThreshold: 50
        livenessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 40
          exec:
            command:
            - "/bin/bash"
            - "-c"
            #- "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi"  //注释
            - "systemctl status glusterd.service"   //添加
          periodSeconds: 25
          successThreshold: 1
          failureThreshold: 50
      volumes:
      - name: glusterfs-heketi
        hostPath:
          path: "/var/lib/heketi"
      - name: glusterfs-run
      - name: glusterfs-lvm
        hostPath:
          path: "/run/lvm"
      - name: glusterfs-etc
        hostPath:
          path: "/etc/glusterfs"
      - name: glusterfs-logs
        hostPath:
          path: "/var/log/glusterfs"
      - name: glusterfs-config
        hostPath:
          path: "/var/lib/glusterd"
      - name: glusterfs-dev-disk
        hostPath:
          path: "/dev/disk"
      - name: glusterfs-dev-mapper
        hostPath:
          path: "/dev/mapper"
      - name: glusterfs-misc
        hostPath:
          path: "/var/lib/misc/glusterfsd"
      - name: glusterfs-cgroup
        hostPath:
          path: "/sys/fs/cgroup"
      - name: glusterfs-ssl
        hostPath:
          path: "/etc/ssl"
      - name: kernel-modules
        hostPath:
          path: "/usr/lib/modules"

3. 設定 Gluster 叢集的描述檔案 topology.json，設定後提供給予 Heketi 使用。

nano topology.json

• 修改項目
• manage: kubernetes 節點名稱
• storage: 節點區域網路 IP
• devices: Data_Disk 在系統上的掛載點
• 修改後貼上

{
"clusters": [
{
  "nodes": [
    {
      "node": {
        "hostnames": {
          "manage": [
            "sdn-k8s-b2-1"
          ],
          "storage": [
            "10.0.0.224"
          ]
        },
        "zone": 1
      },
      "devices": [
        "/dev/sdb1"
      ]
    },
    {
      "node": {
        "hostnames": {
          "manage": [
            "sdn-k8s-b2-2"
          ],
          "storage": [
            "10.0.0.225"
          ]
        },
        "zone": 1
      },
      "devices": [
        "/dev/sdb1"
      ]
    },
    {
      "node": {
        "hostnames": {
          "manage": [
            "sdn-k8s-b2-3"
          ],
          "storage": [
            "10.0.0.226"
          ]
        },
        "zone": 1
      },
      "devices": [
        "/dev/sdb1"
      ]
    }
  ]
}
]
}

設定 Heketi 的 API 資訊驗證的模板

• 複製配置模板

cp heketi.json.template heketi.json

• 編輯配置
  • use_auth: true
  • jwt > admin > key: admin-key
  • jwt > user > key: user-key

{
	"_port_comment": "Heketi Server Port Number",
	"port" : "8080",

	"_use_auth": "Enable JWT authorization. Please enable for deployment",
	"use_auth" : false,

	"_jwt" : "Private keys for access",
	"jwt" : {
		"_admin" : "Admin has access to all APIs",
		"admin" : {
			"key" : ""
		},
		"_user" : "User only has access to /volumes endpoint",
		"user" : {
			"key" : ""
		}
	},

	"_glusterfs_comment": "GlusterFS Configuration",
	"glusterfs" : {

		"_executor_comment": "Execute plugin. Possible choices: mock, kubernetes, ssh",
		"executor" : "${HEKETI_EXECUTOR}",

		"_db_comment": "Database file name",
		"db" : "/var/lib/heketi/heketi.db",

		"kubeexec" : {
			"rebalance_on_expansion": true
		},

		"sshexec" : {
			"rebalance_on_expansion": true,
			"keyfile" : "/etc/heketi/private_key",
			"port" : "${SSH_PORT}",
			"user" : "${SSH_USER}",
			"sudo" : ${SSH_SUDO}
		}
	},

	"backup_db_to_kube_secret": false
}

修復 gk-deploy 版本差異 (ref)

• 修改指令

nano ./gk-deploy 

• 修改功能

刪除參數
--show-all

部署環境

• 準備 kubernetes 環境
  

• 執行部署指令

./gk-deploy -g --admin-key <admin-key> --user-key <user-key> topology.json 

• 獲取 Gluster ID

驗證部署狀態

• Node Label

kubectl get node --show-labels | grep -E “NAME|node”

• GlusterFS 的 DaemonSet Pod

• Heketi 的 Deployment

部署失敗維修

• 獲得 glusterfs DaemonSet Pod 中的版本

kubectl exec <pod> -- glusterfs --version

• pod 一直起不來的話 Not Ready 異常問題(ref)

在異常節點之上
-----
rm -rf /etc/glusterfs /var/lib/glusterd

開搞 Heketi API 的部分進行測試

• 查看 Heketi 狀態

export HEKETI_CLI_SERVER=$(kubectl get svc/heketi --template 'http://{{.spec.clusterIP}}:{{(index .spec.ports 0).port}}')
curl $HEKETI_CLI_SERVER/hello

• 系統回饋

Hello from Heketi

• 設定系統變數 (ref)

echo "export HEKETI_CLI_SERVER=http://$(kubectl get svc heketi -n heketi -o go-template='{{.spec.clusterIP}}'):8080" >> /etc/profile.d/heketi.sh
echo "alias heketi-cli='heketi-cli --user admin --secret <heketi-admin-key>'" >> ~/.bashrc
source /etc/profile.d/heketi.sh
source ~/.bashrc
echo $HEKETI_CLI_SERVER

• 修復一下 Heketi API 權限問題

export HEKETI_CLI_USER=admin
export HEKETI_CLI_KEY=<admin-password>

測試環境

• Heketi 列出叢集資訊

heketi-cli cluster list
heketi-cli cluster info <glusterfs-cluster-id>

• Heketi 列出叢集中的節點/掛載資訊

heketi-cli topology info

• Heketi 列出節點資訊

heketi-cli node list
heketi-cli node info <glusterfs-node-id>

• Heketi 調用 GlusterFS 建立分割區
  • --size=n => n GB
  • --replica=3 => 副本數量

heketi-cli volume create --size=2 --replica=3

• Heketi 觀看分割區狀態

heketi-cli volumn list
heketi-cli volumn info <gluster-volume-id>

• Heketi 刪除分割區

heketi-cli volume delete <gluster-volume-id>

測試環境 on Kubernetes (ref)

• 生成 key

echo -n "mypassword" | base64

• 組建 Secret (使用 key)

apiVersion: v1
kind: Secret
type: kubernetes.io/glusterfs
metadata:
  name: heketi-secret
  namespace: heketi
data:
   # base64 encoded password.
   key: aXRyaS1kZXYtYWRtaW4=

• 部署 Secret 到 kubernetes 之中

kubectl create -f heketi-secret.yaml

• 組建 StorageClass

  • resturl: 系統提供回饋，修改即可使用
  • clusteridL 對應 Gluster 的叢集 id
  • restuser: admin
  • restuserkey: 部署時的 admin-key

• 撰寫 yaml 檔案

storageclass-gluster-heketi.yaml
-----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gluster-heketi-storageclass
parameters:
  resturl: "http://10.100.158.73:8080"
  clusterid: "b3a19fbbde8a5703ec5423fa4745a274"
  restauthenabled: "true"
  restuser: "admin"
  secretName: "heketi-secret"
  secretNamespace: "heketi"
  volumetype: "replicate:3"
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete

• 部署

kubectl apply -f storageclass-gluster-heketi.yaml

• 組建 PVC

pvc-gluster-heketi.yaml 
-----
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-gluster-heketi
spec:
  storageClassName: gluster-heketi-storageclass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi

• 部署

kubectl apply -f pvc-gluster-heketi.yaml 

• 部署使用 GlusterFS 的 Pod

pod-use-pvc-glusterfs-heketi.yaml
-----
apiVersion: v1
kind: Pod 
metadata: 
  name: ubuntu-use-pvc 
spec:
  containers: 
  - name: pod-use-pvc 
    image: ubuntu 
    command: 
    - sleep 
    - "600000"
    imagePullPolicy: IfNotPresent 
    volumeMounts: 
    - name: gluster-volume 
      mountPath: "/testSpeed" 
      readOnly: false 
  volumes: 
  - name: gluster-volume 
    persistentVolumeClaim: 
      claimName: pvc-gluster-heketi 

• 部署

kubectl apply -f pod-use-pvc-glusterfs-heketi.yaml

磁碟壞掉卡住修復

1. 修復法 1 (參考)
2. 修復法 2
   1. kubeadm reset (重置節點)

      kubeadm reset
      

   2. 節點重開機

      init 6
      

   3. 清除硬碟

      wipefs /dev/<mount-point>